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Abstract: 

Command  and  control  (C2)  in  complex,  dynamic,  high-risk  warfighting  environments  is  clearly 
challenging,  particularly  because  of  the  increasing  complexity  of  available  technology  for  processing  and 
presenting  information.  Commanders  need  to  understand  and  act  on  large  volumes  of  information  from  a 
variety  of  sources  and  are  particularly  challenged  by  the  need  to  reason  about  the  qualifiers  of  that 
information,  which  we  will  refer  to  as  meta-information  (e.g.,  uncertainty,  recency,  pedigree).  We  have 
explored  the  role  of  meta- information  in  C2  using  Cognitive  Task  Analysis  (CTA)  techniques  to  identify 
when  and  how,  in  current  practice,  human  interaction  with  meta- information  impacts  decision-making, 
especially  when  that  decision-making  is  supported  by  automation.  Too  often  critical  meta-information  is 
not  processed,  ineffectively  displayed,  or  not  displayed  at  all  in  existing  C2  decision  support  systems.  The 
result  of  our  analyses  is  a  number  of  design  recommendations  for  C2  decision-support  systems  and 
guidelines  for  identifying  and  recognizing  the  need  for  meta-information  processing  and  display.  In  this 
paper,  we  present  the  results  of  our  analyses  and  discuss  their  implications  with  respect  to  the  design  of 
human-system  interfaces  and  the  development  of  computational  information  processing  methods. 

1.  Introduction 

Advances  in  military  command  and  control  (C2)  technologies  have  led  to  increasingly  complex 
systems,  and  a  concomitant  burden  placed  upon  the  commander  to  process  the  information  provided  by 
those  technologies.  A  commander  is  asked  to  process  an  ever-larger  amount  of  information  from  a 
growing  number  of  heterogeneous  sources,  then  compare  that  information  with  past  experiences  and  his 
or  her  knowledge  of  the  current  situation  to  form  a  comprehensive  understanding  of  the  current  situation 
(termed  “situational  awareness”  (Endsley  &  Garland,  2000;  Endsley,  1995).  Given  this  understanding,  a 
commander  is  then  asked  to  make  an  effective  decision.  The  ability  to  make  a  decision  that  will  lead  to 
mission  success  is  largely  predicated  on  the  commander’s  skill  and  experience  in  processing  and 
understanding  information.  This  ability  fundamentally  relies  not  only  understanding  the  domain-related 
information  but  also  the  qualities  of  that  information  (e.g.,  recency,  reliability),  or  the  associated  meta- 
information.  Such  qualities  can  critically  influence  how  a  commander  will  process  information, 
understand  information,  and  make  decisions  based  on  that  information. 

Our  research  across  different  C2  domains  has  yielded  significant  evidence  that  commanders  reason 
using  meta-information.  This  evidence  is  the  result  of  multiple  analyses  of  C2  decision-making  and 
computational  systems  designed  to  support  that  decision-making.  In  some  cases,  meta-information  is  not 
incorporated  into  decision-support  systems  in  those  domains  or  is  inadequately  represented  (both  in  terms 
of  the  underlying  computation  and  in  terms  of  the  displays  and  interfaces  presented  to  the  human  user). 
The  goal  of  this  paper  is  to  describe  our  analytic  efforts  and  describe  the  implications  of  our  analysis  on 
the  development  of  future  C2  systems.  In  this  goal,  we  hope  to  guide  future  C2  research,  design,  and 
development  efforts  to  encourage  the  explicit  incoiporation  of  the  meta-information  critical  to  supporting 
and  enhancing  a  commander’s  reasoning. 

We  begin  by  describing  relevant  background  material,  covering  work  in  related  areas  and  its  impact 
on  our  understanding  of  meta- information’s  influence  on  C2  decision-making  (Section  2).  We  then 
present  our  overall  cognitive  engineering  approach  to  analysis  (Section  3)  including  a  description  of  our 
specific  methods  and  the  application  domains  we  explored  (Section  4).  We  cover  the  results  of  our 
analysis,  focusing  on  defining  meta- information  and  the  sources  and  types  we  encountered  (Section  5). 
Finally,  we  discuss  the  implications  of  these  results  for  the  design  of  C2  decision-support  systems 
(Section  6). 

2.  Background 

Although  there  are  some  prior  references  to  the  term  meta-information  in  the  literature  (Higgins, 
1999),  most  discussions  of  meta-information  relate  to  the  influence  of  uncertainty,  particularly  with 
respect  to  human  decision-making  (Klein,  1996;  Lipshitz  &  Strauss,  1996),  computation  (Halpem,  2003; 
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Parsons,  2001),  and  visualization  design  (McQueary  et  al.,  2004;  Basapur,  Bisantz,  &  Kesavadas,  2003). 
As  we  will  discuss  in  Section  5.1,  our  analyses  have  led  us  to  consider  uncertainty  to  be  one  type  of  meta¬ 
information  and  to  argue  that  a  broader  categorization  is  necessary  to  fully  understand  the  different  types 
and  influences  of  meta-information  on  decision-making.  To  provide  background  for  this  assertion,  we 
present  relevant  background  material  on  the  role  of  uncertainty  in  decision-making,  computational 
reasoning,  and  visualization  below. 

2.1  Uncertainty  and  Human  Decision-Making 

Human  decision-making  under  uncertainty  is  recognized  to  deviate  from  classical,  logical 
decision-making  and  to  be  based  largely  on  experience-based  heuristic  methods  (Kahneman,  Slovic,  & 
Tversky,  1982).  To  better  understand  how  uncertainty  affects  decision-making,  several  attempts  have 
been  made  to  categorize  different  types  of  uncertainty  and  to  identify  how  they  affect  the  decision-making 
process.  One  method  for  classifying  uncertainty  is  to  look  at  its  source;  for  example,  dividing  uncertainty 
into  forms  that  come  from  computational  models  as  opposed  to  a  human  interpretation  (Booker, 
Anderson,  &  Meyer,  2003),  or  using  the  broad  classes  of  physics  uncertainty,  computational  uncertainty, 
visualization  uncertainty,  and  cognitive  uncertainty  (Schunn,  Kirschenbaum,  &  Trafton,  2003).  Another 
method  for  classifying  uncertainty  is  to  look  at  how  it  is  used  in  the  decision-making  process.  This 
method  has  resulted  in  the  development  of  categories  such  as  executional  uncertainty,  goal  uncertainty, 
and  environmental  uncertainty  (Yovits  &  Abilock,  1974).  Another  set  of  classifications  developed  by 
Lipshitz  and  Strauss  (1996),  divides  forms  of  uncertainty  into  inadequate  understanding,  lack  of 
information,  and  conflicted  alternatives.  Similar  taxonomies  were  developed  by  Schunn  et  al  and  Klein 
(1998).  These  classifications  of  uncertainty  have  proven  useful  in  guiding  descriptions  of  how  human 
decision  makers  cope  with  uncertainty.  For  example,  Lipshitz  and  Strauss  (1996)  identify  five  general 
strategies  for  coping  with  uncertainty:  (1)  reducing  uncertainty  by  collecting  more  information,  (2) 
reasoning  by  using  assumptions  to  fill  in  gaps  of  knowledge,  (3)  weighing  pros  and  cons,  (4)  forestalling, 
and  (5)  suppressing  uncertain  information.  Decision-support  systems  can  be,  and  have  been,  designed 
around  supporting  appropriate  strategies  for  dealing  with  uncertainty.  While  these  classifications  of 
uncertainty  and  an  understanding  their  impacts  on  decision-making  have  been  useful  in  the  development 
of  C2  decision  support  system,  they  may  not  generalize  to  other  types  of  meta-information  that  are  not 
fundamentally  based  on  uncertainty  (e.g.,  Do  decision-makers  reason  about  the  recency  of  information  in 
the  same  way  they  do  about  the  certainty  of  information?). 

2.2  Uncertainty  and  Computational  Systems 

Computational  systems  have  been  developed  for  an  enormous  range  of  applications  within  the  C2 
domain.  Inherent  to  the  development  of  such  systems  has  been  the  need  to  reason  computationally  about 
uncertainties  present  in  the  real  world  in  tasks  ranging  from  mission  planning  and  course  of  action 
development  to  mission  execution  and  monitoring  to  after  action  review.  To  support  this  development,  a 
variety  of  computational  approaches  have  been  developed  to  explicitly  support  reasoning  about  one  or 
more  types  of  uncertainty  (Halpem,  2003;  Parsons,  2001).  These  approaches  include:  probability 
measures,  Dempster-Shafer  belief  functions  (Russell  &  Norvig,  2003),  extensions  to  first-order  logic 
(e.g.,  defeasible  reasoning  (McCarthy  &  Hayes,  1969),  argumentation  (Lin,  1993)),  ranking  functions, 
“plausibility”  measures  (Halpem,  2003),  fuzzy  set  theory  (Zadeh,  1965),  and  causal  network  methods 
(e.g.,  Bayesian  belief  networks  (Pearl,  2001;  Pearl  &  Russell,  2000),  similarity  networks  (Geiger  & 
Heckerman,  1996),  influence  diagrams  (Howard  &  Matheson,  1984)).  This  list,  by  no  means  exhaustive, 
represents  the  focus  of  computational  research  on  the  need  to  support  automated  reasoning  about 
uncertainty. 

Some  efforts  have  been  made  within  this  community  to  define  uncertainty  and  to  describe 
taxonomies  of  uncertainties  that  computational  systems  may  reason  about.  Of  these,  Smets  (1997), 
Smithson  (1989),  and  Bose  and  Prade  (1997)  are  notable,  but  it  is  worth  mentioning  that  the  discrepancies 
among  these  taxonomies  would  seem  to  support  Elkan’s  (1994)  assertion  that  developing  such 
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taxonomies  is  largely  a  philosophical  exercise.  Elkan’s  assertion  points  to  the  problems  faced  by  the 
developers  of  such  computational  methods  who  are  not  tightly  tied  to  the  needs  of  human  users  in  a 
particular  application  domain,  where  the  development  of  such  definitions  of  types  of  uncertainty  (and, 
more  generally,  types  of  meta-information)  can  be  achieved  through  formal  knowledge  elicitation 
approaches. 

Relatively  recently,  there  has  been  increased  interest  in  the  management  of  meta-data,  a  term 
used  to  describe  more  broadly  the  various  ways  that  data  may  be  qualified  (Havenstein,  2006;  Marco  & 
Jennings,  2004).  This  term  has  been  applied  to  file  systems,  computer  programs,  images,  relational 
databases,  and  data  warehouses  (i.e.,  its  application  is  largely  contained  within  the  information 
technology  community).  Examples  of  meta-data  include  how,  when,  and  by  whom  a  particular  set  of  data 
was  collected,  and  how  the  data  is  formatted.  This  work  has  been  focused  on  the  tagging  and  handling  of 
data  according  to  its  meta-data,  largely  to  support  system  interoperability  and  search,  rather  than  on 
computational  reasoning  or  computational  intelligence  that  can  be  applied  to  these  data  qualifiers.  Also, 
these  efforts  have  been  focused  on  the  qualities  inherent  in  the  data  rather  than  the  qualities  of  the 
information  that  are  used  by  a  human  decision-maker.  That  is,  they  are  data-centered  rather  than  human- 
centered  in  their  focus. 

2.3  Uncertainty  and  Visualization 

While  some  recent  work  by  Lefevre  et  al.  (2005)  and  Pfautz  and  his  colleagues  (2005a;  2005b; 
2005c)  has  focused  on  the  visual  representation  of  meta-information,  the  majority  of  relevant  literature  is 
focused  on  the  representation  of  one  type  of  meta-information  -  uncertainty  (Trickett  et  al.,  2005; 
McQueary  et  al.,  2004;  Bisantz,  2002;  Barnes,  Wickens,  &  Smith,  2000;  Andre  &  Cutler,  1998;  Pang, 
Wittenbrink,  &  Lodha,  1997a;  Fisher,  1993).  Research  based  in  scientific  visualization  and  geographic 
information  systems  (GIS)  has  explored  graphical  parameters  which  could  be  used  to  encode  the 
uncertainty  in  large  data  sets,  including  attributes  of  scene  geometry  (e.g.,  color,  shading,  and  bumpiness 
(Pang,  Wittenbrink,  &  Lodha,  1997b))  or  traditional  graphic  variables  used  in  cartography  (e.g.,  texture, 
color,  orientation,  and  shape  (MacEachren,  1992)).  Pang  et  al.  (1997b)  suggested  the  use  of  glyphs 
(graphical  forms  such  as  arrows  or  vertical  lines),  in  part  because  it  frees  other  graphical  dimensions  for 
other  purposes.  Glyphs  have  previously  been  used  to  represent  magnitude  and  direction  of  winds  and 
ocean  currents  along  with  the  uncertainties  in  these  dimensions.  In  one  example,  the  general  shape  of  the 
glyph  was  an  arrow  with  the  width  of  the  arrowhead  represented  uncertainty  in  heading,  while  multiple 
arrowheads  represented  uncertainty  in  magnitude  (Lodha  et  al.,  1996).  Lodha  et  al.  (1996)  claim  that  the 
use  of  such  visualizations  results  in  an  integrated  graphic  “so  that  users  cannot  help  but  interpret  the 
resulting  image  holistically.”  Therefore,  these  graphics  can  be  seen  as  examples  of  object  displays,  which 
have  been  proposed  for  the  integrated  display  of  system  information  (Carswell  &  Wickens,  1996; 
Bennett,  Toms,  &  Woods,  1993).  Other  suggested  techniques  have  included  pairs  of  graphics  showing  a 
value  and  corresponding  uncertainty  (simultaneously  or  alternately  (MacEachren,  1992)),  animation  (via 
the  degree  of  motion),  and  sound  (Pang  et  al.,  1997b).  However,  such  representations  have  not  been 
systematically  evaluated.  In  all  of  these  cases,  no  work  has  been  done  to  extend  our  understanding  of 
how  such  representations  might  be  applied  to  different  types  of  meta-information  (or,  for  that  matter,  even 
different  types  of  uncertainty). 

3.  Method  -  Cognitive  Systems  Engineering 

Our  goal  across  the  multiple  projects  discussed  in  this  paper  has  been  to  identify  and  study  types 
of  meta- information  and  their  influences  on  decision-making  with  the  express  purpose  of  aiding  in  the 
design  of  new  C2  decision-support  systems.  To  accomplish  this  goal,  we  chose  a  particular  set  of  analytic 
methods  encompassed  as  part  of  Cognitive  Systems  Engineering  (CSE).  CSE  was  developed  as  a  result 
of  experiences  with  new  technology  where  increased  computerization  did  not  guarantee  improved  overall 
human-and-computer  system  performance  and,  in  some  cases,  led  to  catastrophic  errors  (e.g.,  confusions 
leading  to  pilot  error  and  fatal  aircraft  accidents)  (Woods  &  Dekker,  2000;  Roth,  Malin,  & 
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Schreckenghost,  1997;  Woods,  Sarter,  &  Billings,  1997).  Cognitive  Systems  Engineering  (CSE)  attempts 
to  prevent  these  types  of  failures  in  the  design  and  development  of  complex  systems  by  addressing  design 
issues  through  careful  analysis  of  the  problem  domain,  the  tasks  to  be  performed  by  a  human-computer 
system,  and  the  limitations  of  both  the  human  and  the  machine. 

The  field  of  CSE  encompasses  a  variety  of  analysis  and  design  methods  (e.g.,  (Bums  & 
Hajdukiewicz,  2004;  Eggleston,  Roth,  &  Scott,  2003;  Elm  et  al.,  2003;  Endsley,  Bolte,  &  Jones,  2003; 
Vicente,  1999;  Rasmussen,  Pejtersen,  &  Goodstein,  1994)).  These  methods  share  a  commitment  to 
analyzing  the  cognitive  and  collaborative  demands  imposed  by  the  domain  of  practice  and  identifying 
implications  for  information,  visualization,  and  decision-support  requirements.  CSE  methods  generally 
entail  a  multi-phase,  iterative  design  approach  that  includes  a  cognitive  analysis  phase,  a  concept 
development  and  prototyping  phase,  and  a  user  evaluation  phase.  The  cognitive  analysis  phase  typically 
employs  knowledge  elicitation  methods  such  as  interviews  of  domain  practitioners  and  observations  of 
work  in  context.  These  methods  uncover  the  cognitive  and  collaborative  processes  involved  in  making 
decisions  and  performing  tasks  in  the  domain  and  the  challenges  that  arise  (Potter,  Roth,  Woods  and  Elm, 
2000;  Roth  and  Patterson,  2005).  A  variety  of  representational  formalisms  are  used  to  synthesize  and 
communicate  the  results  of  the  analysis  and  the  design  implications.  These  formalisms  range  from  graphic 
representations  such  as  goal-means  graphs,  to  structured  tables,  to  prose  summary  descriptions  (Elm, 
Potter,  Gualtieri,  Roth  &  Easter,  2003;  Eggleston,  Roth,  Whitaker  and  Scott,  2005;  Endsley  et  al,  2003; 
Vicente,  1999;  Wampler,  Whitaker,  Roth,  Scott,  Stilson  &  Thomas-Meyers,  2005).  The  cognitive  analysis 
phase  is  followed  by  a  concept  development  phase  that  involves  identifying  aiding  concepts  to  address  the 
challenges  identified  by  the  cognitive  analysis,  and  developing  storyboards  and  prototypes  to  realize  those 
concepts.  The  prototype  is  then  tested  in  the  user  evaluation  phase.  Experienced  domain  practitioners  are 
the  test  participants.  They  assess  the  viability  of  the  aiding  concepts  and  drive  further  design.  CSE  is  an 
iterative  process  that  typically  involves  multiple  iterative  loops  through  the  cognitive  analysis,  concept 
development,  and  user  evaluation  phases.  This  cyclic  process  is  illustrated  in  Figure  3-1. 


Analysis  Iterative  Development  Deployment 


Figure  3-1:  Cognitive  Engineering  approach  to  C2  decision-support  system  design, 

development,  and  evaluation 

The  nature  of  military  decision-making  analysis  tasks  in  the  face  of  increased  information  flow 
suggests  that  the  design  and  development  of  tools  to  support  decision-making  will  be  greatly  aided  by  the 
adoption  of  a  CSE  approach.  This  approach  incorporates  an  understanding  of  the  problem  domain  and  its 
complexities,  an  analysis  of  the  functional  demands  on  the  human  user  and  any  support  tools,  and  an 
analysis  of  human  perceptual  and  cognitive  limitations  in  the  context  of  a  realistic  set  of  case  studies,  and 
so  provides  a  significant  amount  of  highly  relevant  information  to  the  system  designer  as  regards  the 
interdependencies  between  information  and  decision-making.  Because  of  these  qualities,  we  have  chosen 
it  as  our  fundamental  methodological  framework  for  the  study  of  meta-information  effects  on  decision¬ 
making.  In  addition,  the  CSE  approach  allows  for  a  spiral  design  process,  whereby  initial  designs  can  be 
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evaluated  with  expert  analysts  to  see  how  effectively  each  design  supports  the  decision-making  process. 
We  and  our  colleagues  have  used  this  approach  for  a  wide  variety  of  projects  (Roth  &  Patterson,  2005; 
Pfautz  et  al,  2004;  Roth  et  al.,  2002;  Grecu,  Sullivan,  &  Zacharias,  2001;  Das  et  al.,  2000)  and  its  use  is 
espoused  throughout  the  system  engineering  community  (Eggleston,  2003;  Woods  &  Roth,  1988). 

4.  Analysis 

In  the  process  of  designing  and  developing  a  variety  of  systems  to  support  different  aspects  of  C2 
across  different  application  domains,  we  applied  the  CSE  methodology  and  discovered  a  number  of 
consistencies  regarding  the  types  and  sources  of  information  and  meta-information  about  which  a 
commander  may  reason.  Below,  we  discuss  the  problem  domains  studied,  our  specific  analytic  goals, 
and  the  nature  of  the  work  performed. 

As  a  team,  we  have  applied  CSE  methods  to  system  design  and  development  in  a  wide  variety  of 
domains,  from  Air  Force  mission  management  and  planning  tools  to  Army  Intelligence  collection  tools 
(Wampler  et  al.,  2005;  Pfautz  et  al.,  2005c).  In  this  paper,  we  are  consider  only  the  work  performed  to 
understand  C2  decision-making  and  the  impacts  of  various  types  of  C2  decision-support  systems, 
including  analyses  performed  across  the  domains  shown  in  Table  4-1 


Table  4-1:  Domains  studied  in  analysis  efforts 


Intelligence  operations 

We  analyzed  the  planning  and  collection  of  Army  intelligence  to  help 
develop  tools  to  support  collection  planning  and  information  fusion.  For 
example,  we  studied  how  commanders  and  intelligence  analysts  reason 
about  how  the  credibility  of  a  source  of  intelligence  interacts  and  the 
confidence  stated  by  that  source  (Pfautz  et  al.,  2004). 

Small-unit  tactical  maneuvers 

We  investigated  how  platoon-level  Army  commanders  plan  and  execute 
tactical  maneuvers  in  urban  and  open-field  terrain  to  support  the 
development  of  systems  to  aid  in  training  spatio-temporal  reasoning 
skills.  For  example,  a  commander  may  reason  about  the  recency  of 
information  about  downed  power  lines  along  a  key  corridor  to  determine 
if  it  is  trafficable  for  large  vehicles. 

Sensor  management 

We  studied  the  planning  and  collection  of  intelligence  using  relatively 
novel  sensor  field  technologies.  In  this  domain,  a  commander  needs  to 
reason  about  the  “health”  of  the  source  and  the  frequency  of  data 
collection  (Pfautz  et  al.,  2005c) 

Weather  impact  assessment 

We  studied  C2  decision-making  in  Army  and  Air  Force  missions  to  help 
develop  systems  that  help  commanders  and  their  staff  reason  about  the 
influence  of  weather  events  on  tactical  operations.  For  example,  a 
commander  may  reason  about  the  timing  and  severity  of  an  incoming 
storm  to  determine  whether  or  not  to  send  a  UAV  on  a  reconnaissance 
mission  (Lefevre  et  al.,  2005;  Wampler  et  al.,  2005) 

Natural  disaster  management 

We  explored  how  firefighters  were  managed  and  directed  during  wildfire 
events  in  the  Western  U.S.  to  support  the  development  of  decision-aids 
for  reasoning  about  asset  allocation.  For  example,  a  decision-maker 
might  reason  about  the  validity  of  a  firespread  model  in  determining 
whether  or  not  to  evacuate  a  nearby  town  (Pfautz  et  al.,  2005a). 

All  of  these  domains  involve  a  commander  or  decision-maker  who  must  reason  about  incoming 
information  and  its  associated  meta-information.  While  this  is  not  a  complete  set  of  possible  C2 
application  domains,  we  believe  that  it  represents  a  sufficiently  broad  set  to  identify  particular 
commonalities  across  the  types  of  information  and  meta- information  required  to  make  effective  decisions. 

In  each  of  the  above  domains,  we  performed  an  analysis  to  identify  characteristics  of  the  domain  and 
decision-making  in  the  domain  that  had  important  implications  for  system  design.  We  adapted  CSE 
methods  such  as  Cognitive  Task  Analysis  (CTA),  and  while  we  were  focused  on  the  features  of  a 
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particular  domain,  we  also  established  some  common  analytic  goals  across  domains.  These  goals 
included  identifying: 

■  Key  decisions  and  decision  types 

■  Sources  and  types  of  data  informing  those  decisions 

■  Sources  and  types  of  meta-information  influencing  those  decisions 

■  Contextual  (e.g.,  situational,  environmental)  factors  influencing  information  and  meta¬ 
information  needs 

The  analyses  were  based  on  structured  interviews  with  a  number  of  experienced  experts  in  each 
of  the  domains  listed  in  Table  4-1.  Two  converging  techniques  were  used  to  elicit  information  on  the  role 
of  meta- information  in  complex  decision-making  situations.  One  technique  relied  on  critical  incident 
analysis.  In  this  case,  the  domain  experts  were  asked  to  describe  actual  military  situations  they 
experienced  in  which  they  faced  cognitively  demanding  decisions.  A  second  technique  involved  the 
development  and  analysis  of  hypothetical  scenarios  that  characterized  decision-making  demands  in 
complex  situations  involving  sparse,  noisy,  conflicting,  and  ambiguous  data.  The  scenarios  were 
typically  designed  in  collaboration  with  the  domain  experts  that  participated  in  the  analysis  and  provided 
a  vehicle  that  enabled  the  domain  experts  to  concretely  articulate  the  kinds  of  complexities  that  arise  in 
the  domain  that  pose  challenges  to  cognitive  and  decision-making  processes.  The  exercise  of  developing 
the  scenarios  and  analyzing  the  potential  sources  of  complexities  that  would  be  imposed  on  decision¬ 
makers  enabled  the  domain  experts  to  expose  knowledge  of  the  domain  and  sources  of  complexity  that 
would  otherwise  be  difficult  for  them  to  articulate.  In  some  cases,  the  hypothetical  scenario  was 
presented  to  one  or  more  additional  domain  experts  to  elicit  additional  perspectives  on  what  made  these 
situations  challenging  and  how  experts  and  less  experienced  operational  personnel  would  respond  in  these 
complex  decision-making  situations.  Given  these  scenarios,  the  domain  experts  were  interviewed  about 
their  decisions  at  various  points  in  the  scenario.  Specific  implications  of  decisions  were  studied,  with 
particular  regard  to  the  types  of  conditions  and  factors  impacting  on  the  decision-making  process.  In  each 
domain,  a  set  of  cognitive  tasks  was  identified,  along  with  particular  cognitive  challenges  arising  in  the 
performance  of  those  tasks.  In  this  paper,  we  are  presenting  the  results  from  our  analysis  of  the 
challenges  related  to  the  processing  and  integration  of  meta-information. 

5.  Results 

Our  experience  performing  analyses  of  decision-making  across  different  C2-related  domains  led 
us  to  recognize  some  common  concepts  and  issues  related  to  meta-information.  As  part  of  our  analysis, 
we  developed  some  general  concepts  about  what  should  be  designated  as  “meta-information”  and  how  we 
might  define  types  of  meta-information  in  a  particular  domain.  In  addition,  we  developed  a  categorized 
list  of  types  of  domain-specific  meta-information  we  discovered,  which  may  aid  in  the  development  of 
meta-information  needs  for  other  domains.  Finally,  we  identified  some  key  characteristics  of  how 
commanders  reason  about  meta- information,  therefore  supporting  the  development  of  future  C2  decision- 
support  systems. 

5.1  General  Meta-Information  Concepts 

Before  we  can  develop  an  understanding  of  what  qualifies  information  (i.e.,  What  is  meta¬ 
information?),  we  need  to  distinguish  between  data  and  information.  The  distinction  between  data 
(minimally  processed,  time-sequenced  output  from  sources  such  as  sensors,  humans,  or  algorithms)  and 
information  (in  which  data,  often  from  multiple  sources  or  times,  is  integrated  based  on  the  needs  of 
human  decision  makers)  is  common  within  the  field  of  Cognitive  Engineering.  Elm  et  al.  (2003)  describe 
a  “data-to-information”  transition  as  part  of  their  cognitive  engineering  methodology.  Specifically,  this 
transition  transforms  parameters  (i.e.,  based  on  sensed  data  and  computation)  into  information  that  is 
relevant  to  operator  goals  (e.g.,  whether  a  parameter  has  exceeded  a  limit).  Elm  et  al.  provide  several 
rules-of-thumb  for  distinguishing  information  requirements  from  data,  including  “if  you  are  listing 
sensors,  you  are  wrong,”  “does  it  have  an  intent-functional  ‘feel’  to  it,”  and  “expect  phrases  and 
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sentences,  not  database  tags.”  Endsley  et  al.  (2003)  claim  that  information  requirements  (e.g.,  “enemy 
strengths”)  should  be  identified  without  reference  to  the  technology  with  which  the  information  is 
obtained,  thus  implicitly  separating  information  from  the  data  sources  which  provide  information. 

Following  these  distinctions,  we  differentiate  between  data  (the  outputs  from  human  and  non¬ 
human  sensors  or  algorithms)  and  functional  information  (information  that  is  relevant  within  a  goal- 
directed,  decision-making  process).  Functional  information  requirements  are  those  requirements 
identified  through  methods  such  as  cognitive  task  or  work  analysis  and  are  clearly  tied  to  the  goals  and 
tasks  of  operators.  Meta-data  and  meta-information  are  characteristics  or  qualifiers  of  data  and 
information  in  some  particular  context,  respectively.  For  instance,  if  data  corresponds  to  the  signals 
transmitted  from  a  sensor,  then  meta-data  may  “tag”  the  signals  with  temporal  and  location  information, 
along  with  measurements  of  the  sensor’s  power  or  known  accuracy.  Environmental  characteristics,  such 
as  ambient  temperature  or  humidity  may  also  be  included  as  meta-data.  As  discussed  earlier,  meta-data 
may  be  used  in  a  variety  of  computational  systems  for  filtering,  managing,  and  processing  data  before  it 
becomes  an  input  to  a  decision-making  process.  It  can  be  inherent  in  the  data  or  calculated  separately. 
Meta-information  can  be  thought  of  as  the  qualifications  or  characteristics  of  that  functional  information 
that  allows  operators  to  correctly  interpret  that  functional  information  as  needed  (e.g.,  this  information  is 
too  old  to  be  pertinent  to  my  current  situation).  Meta-information  includes  uncertainty  but  should  not  be 
thought  of  as  a  way  of  describing  different  types  of  uncertainty.  For  example,  a  commander  may  have  as 
a  task  to  read  a  list  of  reports.  This  rank-ordered  list  of  reports  can  be  read  without  any  uncertainty  about 
this  task.  However,  if  the  commander’s  intelligence  officer  tells  him  a  different  ordering,  for  example, 
that  a  previously  low-ranked  report  is  important  and  will  impact  his  ability  to  accurately  interpret  other 
reports,  then  the  commander  could  re-order  the  reports  and  process  them  differently  because  of  this  meta¬ 
information.  In  this  example,  we  might  argue  that  there  is  no  uncertainty  about  the  order  of  the  reports, 
only  a  difference  in  the  meta-information  the  commander  has  to  decide  the  order  in  which  to  read  them. 


Figure  5-1:  Distinguishing  between  data,  meta-data,  information,  and  meta-information 

As  shown  in  Figure  5-1,  functional  information  and  meta-information  are  derived  by  the  human 
user  from  the  transformation  and  integration  of  data  and  meta-data.  For  the  purposes  of  designing 
decision-support  systems,  the  distinction  between  information  and  data  is  advantageous,  because  it  allows 
the  identification  of  new  requirements  for  data  or  meta-data  by  allowing  the  specification  of  decision-, 
goal-,  or  task-relevant  information  which  may  be  currently  unavailable  leaving  only  sensor  or  processing 
capabilities  to  guide  system  design.  This  distinction  allows  us  to  differentiate  between  what  needs  to  be 
known  by  the  commander,  and  the  data  and  computational  methods  that  are  available  to  provide  that 
information.  For  instance,  within  this  framework  it  is  conceivable  to  identify  information  or  meta¬ 
information  needs  that  cannot  be  met  with  currently  available  data  or  meta-data  and  associated  processing 
methods,  or  those  which  may  be  provided  with  higher  quality  as  more  data  and  more  sophisticated 
methods  for  combining  the  data  become  available  (Llinas  et  al.,  2004).  The  identification  of  such  needs  is 
cited  as  a  typical  outcome  of  a  cognitive  engineering  analysis  (Elm  et  al.,  2003;  Vicente,  1999).  Also,  it  is 
likely  that  the  types  of  meta-data  that  could  be  associated  with  any  data  source  (particularly  when 
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environmental  information  is  included)  are  numerous  to  the  point  of  being  impractical  (to  define,  identify, 
store,  and  process).  Taking  a  top-down  approach  based  on  human-centered  information  and  meta¬ 
information  needs  can  serve  as  a  constraint  on  the  types  of  meta-data  that  need  to  be  considered  within  a 
computational  decision-support  system  (i.e.,  identified,  calculated,  tagged,  etc.). 

The  types  of,  and  distinctions  between,  information  and  meta- information  are  based  on  the  task  or 
goal  being  pursued,  as  illustrated  by  the  example  in  Table  5-1.  This  example  is  taken  from  two  tasks  in  a 
typical  military  C2  domain:  sensor  management  and  tactical  decision-making  that  is  dependent  on  sensor 
data. 

Table  5-1:  Examples  illustrating  the  task-dependency  of  information  and  meta-information 

specification 


Sensor  Management 

Tactical  Decision-Making 

Data 

Sensor  X  reports  42.2  dB 

Sensor  Y  reports  32.1°  F 

Sensor  X  reports  42.2  dB 

Sensor  Y  reports  32.1°  F 

Meta-Data 

Sensor  X  error  is  ±  .4  dB 

Sensor  Y  reports  at  5  Hz 

Sensor  X  error  is  ±  .4  dB 

Sensor  Y  reports  at  5  Hz 

Functional 

Information 

Location  of  sensors 

Sensor  types 
“Health”  of  sensors 

Location  of  targets 

Type  of  targets 

Number  of  targets 

Meta-Information 

Accuracy  of  sensor  status 

Recency  of  sensor  status 

“Health”  of  sensors 

Coverage  of  sensors 

Accuracy  of  target  information 
Recency  of  target  information 

Decision-Making  Impact  of 
Meta-Information 

Effectiveness  of  sensor- 
relocation  or  employment 
decisions 

Satisfaction  of  targeting  policy 
factors  (Rules  of  Engagement) 

Note  first  that  for  both  tasks,  data  and  meta-data  types  are  the  same  and  are  obtained  directly  from 
the  sensors.  Data  is  defined  by  the  inherent  reporting  capabilities  of  a  sensor  or  sensor  fusion  system.  The 
meta-data  in  this  example  could  be  provided  by  the  specifications  of  the  sensor  or  by  some  calculation  of 
average  error  or  reporting  frequency.  It  is  when  considering  information  and  meta-information  that  the 
difference  between  tasks  becomes  apparent.  The  sensor  management  task  includes  decisions  about 
whether  to  remove  the  sensors,  replace  the  sensor,  or  reconfigure  the  sensors.  In  this  task,  the  status  of  the 
sensors  is  the  functional  information,  and  the  meta-information  concerns  the  accuracy  and  recency  of  that 
status  information.  In  the  tactical  decision-making  task,  where  a  commander  will  interpret  and  use 
information  about  the  operational  environment  to  direct  his  or  her  assets,  the  functional  information  is  the 
location  and  type  of  objects  of  interest,  while  the  meta- information  includes  the  status  of  the  sensors 
providing  this  information. 

As  a  result  of  our  analysis,  we  argue  that  simply  considering  the  role  of  uncertainty  in  decision¬ 
making  may  be  insufficient  to  fully  understand  the  reasoning  process  and  that  additional  types  of 
information  qualifiers  should  be  considered.  We  contend  that  there  are  many  different  types  of 
information  qualifiers  that  may  be  termed  “meta-information”  and  that  when  building  decision-aids  it  is 
useful  to  understand  the  various  kinds  of  meta-information  that  decision-makers  use  in  their  reasoning. 
To  support  these  assertions,  we  have  developed  working  definitions  for  data,  meta-data,  information  and 
meta-information  (adapted  from  Pfautz  et  al.,  2005  c): 

■  Data  is  output  (processed  or  unprocessed)  from  a  human,  machine,  or  human-machine  system  (e.g., 
John  said  it  is  raining,  acoustic  sensor  X  reported  34  dB,  Bill  reported  his  GPS  location  is 
44°23’ 13.02”) 
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■  Information  (or  functional  information)  is  an  input  to  a  goal-directed  decision-making  process 
(e.g.,  because  information  has  been  received  that  there  are  high-intensity  crosswinds  at  the  launch 
site,  the  commander  decides  the  aircraft  will  not  be  flown) 

■  Meta-data  is  a  characteristic  or  qualifier  of  data  (e.g.,  John  has  bad  eyesight,  acoustic  sensor  Y  has 
an  error  of  ±  0.2  decibels,  Bill  reported  his  location  at  1400) 

■  Meta-information  is  a  characteristic  or  qualifier  of  information  that  affect  a  commander’s: 

■  Information  processing  (e.g.,  reports  flagged  as  “important”  by  an  aide  get  read  first) 

■  Situational  awareness  (e.g.,  because  information  about  adversary  surface-to-air  missile  locations 
is  recent  and  certain,  the  commander  knows  certain  friendly  aerial  assets  are  under  threat) 

■  Decision-making  (e.g.,  because  information  about  the  adversary’s  location  is  30  hours  old,  the 
commander  decides  to  send  a  scout  to  confirm  the  location  before  maneuvering) 

As  described  above,  what  is  defined  as  information  and  meta-information  in  a  particular  situation  is 
inherently  subject  to  the  type  of  task  being  performed.  It  is  necessary  to  understand  a  commander’s  tasks, 
goals,  or  decisions  before  these  definitions  can  be  applied  and  used  to  guide  system  design.  We  believe 
these  definitions  present  a  generalizable  view  of  meta-information  and  therefore  capture  important  meta¬ 
information  that  cannot  be  classified  as  a  type  of  uncertainty. 

5.2  Sources  and  Types  of  Meta-Information 

A  main  objective  of  our  analyses  was  to  define  and  characterize  types  of  meta- information  that 
influence  military  decision-makers  across  a  wide  range  of  C2  situations.  Our  goal  was  to  provide  a  broad 
and  principled  characterization  of  the  range  of  meta-information  that  may  need  to  be  included  in  C2 
decision-support  systems  to  provide  effective  support.  Through  our  analysis  efforts,  we  identified  the 
main  types  of  decisions  (and/or  tasks  and  goals)  and  the  types  of  meta-information  that  impact  those 
decisions  in  the  specific  domains  we  studied.  As  described  above,  the  specific  aspects  of  meta¬ 
information  that  are  (or  should  be)  considered  by  the  decision-maker  depend  on  the  particular  domain  of 
application.  By  examining  the  specific  factors  that  contribute  to  and  constitute  meta-information  in  the 
multiple  domains  we  examined,  we  were  able  to  define  a  list  of  specific  types  of  meta-information  we 
encountered.  We  list  and  describe  these  meta- information  types  in  Table  5-2  with  the  intent  to  describe 
examples  of  meta-information  discovered  in  particular  domains.  These  types  are  not  intended  to  be 
universal  across  types  of  tasks/decisions/goals,  nor  across  different  application  domains.  Rather,  it  is 
intended  to  aid  in  the  identification  of  similar  types  of  meta-information  when  analyzing  different  types  of 
decision-making  in  other  domains. 


Table  5-2:  Sources  and  types  of  meta-information  in  the  explored  domains 


Meta-Information  Type 

Sub-types  or  related  types 

Characteristics  of  the  source  of  information 

Type  of  data  the  source  can  produce 

Type  of  processing  used 

Range  of  data  generated 

Baseline  error  rates 

Frequency  of  reporting 

Ability  to  report  on  its  status  and  characteristics  of 
that  report 

Inherent  biases 

Past  performance,  history 

Directly  observing  or  deriving  information 

Characteristics  of  the  source  as  a  function  of 
other  factors 

Time 

Location  in  environment  (e.g.,  terrain,  weather) 

Types  of  intermediate  processing 

Content  of  report 
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Information  context 

Uncertainty 

Spatial  uncertainties 

Temporal  uncertainties 

Uncertainties  about  uncertainty  reporting 

Likelihood 

Probability 

Confidence 

Accuracy 

Precision 

Ambiguity 

Specificity  or  resolution  of  information 

Level  of  abstraction  of  information 

Information  context 

(i.e.,  relationship  to  other  information) 

Degree  of  confirming  or  disconfirming  information 
Paucity  of  information 

Frequency  of  reporting  of  information 

Missing  or  degraded  information  qualifiers 
Information-to-noise  ratio 

History 

Reliability  of  source 

W.r.t.  source  characteristics 

W.r.t.  information  context 

Credibility  of  content  from  source 

W.r.t.  reliability 

W.r.t.  type  of  content 

W.r.t.  type  of  source 

W.r.t.  information  context 

Relevance  or  pertinence 

W.r.t.  specific  mission  goals 

W.r.t.  actual/perceived  information  needs 

W.r.t.  broader  operational  context 

W.r.t.  current  hypotheses  about  the  situation 

Temporal  qualifiers 

Staleness 

Recency 

Certainty  about  time  of  reporting 

Latency 

Lag 

Absence  of  expected  information 

This  list  of  types  of  meta-information  represents  the  types  discovered  in  the  domains  we  studied.  There 
are  clearly  interactions  between  types  and  sub-types  of  meta-information  that  deserve  further  study  across 
additional  domains  and  decision-making  tasks.  There  are  also  myriad  semantic  issues  with  respect  to  how 
these  types  of  meta-information  should  be  defined  (e.g.,  how  should  we  differentiate  between  types  of 
relevance  meta-information?).  As  such,  our  initial  goal  is  to  simply  establish  the  need  for  analysis  of 
meta-information  needs  in  the  particular  context  of  a  decision-making  task  and  to  provide  a  starting  point 
for  understanding  what  types  of  needs  might  be  identified. 

5.3  Meta-Information  and  Decision-Making 

The  number  of  qualifiers  that  can  impact  data  interpretation  and  utilization,  and  the  need  to 
integrate  multiple  qualifiers,  makes  clear  the  cognitive  complexity  faced  by  military  decision-makers  as 
they  try  to  interpret  the  relevance  and  importance  of  any  given  piece  of  data  to  their  decision-making 
process  and  to  integrate  across  multiple  pieces  of  information  to  draw  inferences  and  conclusions.  The 
difficulties  faced  by  a  decision-maker  could  be  characterized  as  more  generalized  versions  of  the  five 
cognitive  strategies  identified  by  Lipshitz  et  al.  (1996)  for  coping  with  uncertainty  (see  Section  2.1).  In 
our  research,  we  have  identified  three  main  cognitive  complexities  relating  to  meta-information.  First,  the 
decision-maker  may  fail  to  recognize  relevant  meta-information.  For  example,  the  commander  may  fail 
to  appreciate  that  a  message  received  from  headquarters  is  old  (i.e.,  stale)  and  therefore  may  no  longer  be 
true.  Second,  the  commander  may  not  process  meta-information  appropriately.  That  is,  the  analyst  may 
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fail  to  correctly  integrate  reliability  and  credibility  of  a  message  to  come  up  with  an  overall  confidence  in 
that  message.  Third,  the  commander  may  not  properly  utilize  the  meta-information  to  integrate  multiple 
diverse  information  reports  that  have  different  meta- information  qualifiers  (e.g.,  different  sources, 
different  levels  of  reliability,  different  levels  of  credibility,  different  time  stamps,  different  perceived 
relevance  to  the  current  mission).  There  is  also  evidence  that  humans  have  difficulty  appropriately 
weighing  and  integrating  multiple  information  sources  that  vary  on  meta-information  dimensions 
(Wickens,  Pringle,  &  Merlo,  1999),  further  increasing  the  commander’s  burden. 

6.  Implications  and  Discussion 

We  found  that  a  commander’s  decision-making  performance,  situational  awareness,  workload, 
and  trust  can  be  influenced  by  the  inclusion  of  meta-information  in  a  system’s  display,  interface,  and 
underlying  computational  formalisms.  The  results  of  our  analyses  point  to  implications  for  the 
development  of  C2  decision-support  tools.  These  implications  affect  specific  components  of  decision 
support  systems,  for  example,  display  design  (e.g.,  present  needed  meta- information,  but  only  in  specific 
situations  to  avoid  overload),  user  interaction  design  (e.g.,  allow  the  human  user  to  control  when  and  what 
meta-information  is  presented),  and  automation  design  (e.g.,  use  computational  methods  that  can  provide 
meta-information  about  both  the  actions  of  the  automated  processes  and  their  results). 

6.1  Implications  for  Computational  System  Design 

The  need  for  meta-information  impacts  the  design  of  computational  decision-support  systems  at 
many  levels.  Ideally,  the  need  for  meta-information  should  impact  the  development  of  sensors  and 
processing  systems  that  can  address  those  specific  needs.  More  realistically  (and  because  system 
engineering  is  often  driven  more  by  technical  and  fiscal  constraints  than  by  human  user  needs),  systems 
need  to  be  developed  that  can  map  existing  data  and  meta-data  to  specific  meta- information  needs.  There 
are  a  number  of  technologies  from  the  data  fusion  community  that  could  be  retroactively  adapted  to 
provide  meta- information  to  address  the  failures  of  the  human-computer  system  to  recognize,  process, 
and  utilize  meta-information.  Similarly,  there  are  techniques  in  the  computational  intelligence 
community  that  could  be  easily  adapted  to  calculating  different  types  of  meta- information.  For  example, 
Bayesian  networks  could  be  used  to  calculate  a  “confidence”  value  based  on  combining  different  meta¬ 
information  types  such  as  information  context  (e.g.,  degree  of  confirming/disconfirming  evidence),  source 
characteristics  (e.g.,  sensor  error  rates),  and  the  interaction  of  the  source  characteristics  with  the 
environment  (e.g.,  error  rates  as  a  function  of  temperature).  From  a  purely  theoretical  basis,  there  is  little 
reason  why  meta- information  could  not  be  calculated  for  many  problem  domains.  However,  the 
calculation  of  meta-information  can  represent  an  additional  computational  burden  and  could  decrease  the 
efficiency  of  some  systems.  Ideally,  meta-information  calculation  methods  that  can  be  incorporated  with 
existing  methods  for  generating  needed  information  or  that  can  be  used  selectively  and/or  on-demand 
should  be  developed. 

6.2  Implications  for  Display  and  Interface  Design 

In  the  Cognitive  Systems  Engineering  approach  that  we  have  employed  in  this  work,  the  clear 
identification  of  information  and  meta-information  needs  leads  not  only  to  the  design  of  computational 
methods  to  address  these  needs  but  also  to  the  development  of  displays  and  interfaces  that  can  effectively 
communicate  the  needed  information  and  meta- information  to  the  commander  in  ways  that  specifically 
address  possible  failures  to  recognize,  process,  and  utilize  meta- information.  The  knowledge  that  humans 
have  difficulty  appropriately  weighing  and  integrating  multiple  information  sources  that  vary  on  meta¬ 
information  dimensions  (Wickens  et  al.,  1999)  highlights  the  need  to  (1)  more  effectively  communicate 
the  meaning  and  basis  (i.e.,  rationale)  for  how  and  why  information  was  produced  and  provide  (2) 
intelligent  support  for  integrating  and  reasoning  about  meta-information  with  respect  to  both  a  given  piece 
of  information  and  to  multiple  pieces  of  information  that  differ  along  meta-information  dimensions  (e.g., 
differ  in  reliability  and  credibility).  At  the  same  time,  our  analysis  and  the  results  of  developing 
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preliminary  display  designs  suggests  that  presentation  of  meta-information  needs  to  be  context  sensitive 
(Lefevre  et  al.,  2005;  Pfautz  et  al.,  2005a;  Pfautz  et  al.,  2005c).  Presentation  of  meta-information  under 
all  circumstances  can  contribute  to  information  overload  conditions  and  result  in  failures  to  recognize 
and/or  use  meta-information  in  reasoning.  Therefore,  it  is  important  to  examine  the  circumstances  under 
which  decision-makers  will  be  critically  affected  by  the  presence  or  absence  of  meta- information. 
Situations  where  there  are  “close  calls”  or  where  the  risk  associated  with  alternative  actions  is  high  are 
likely  to  require  more  careful  consideration  of  meta-information  in  decision-making  to  improve  the 
salience  of  important  information  while  avoiding  information  overload. 

Our  results  point  to  additional  guidelines  for  display  design.  While  some  work  has  been  done  on 
the  visualization  of  uncertainty  (Section  2.3),  comparatively  little  research  has  been  done  to  explore  the 
more  general  problem  of  establishing  effective  forms  for  communicating  meta-information.  Clearly,  an 
important  constraint  is  avoiding  the  presentation  of  too  much  information  in  conflicting  forms,  which 
could  easily  occur  if  meta-information  is  displayed  without  thought  to  how  it  might  optimally  compliment 
the  information  it  is  qualifying.  Another  approach  is  to  provide  methods  for  user-selected  display  of 
meta-information,  although  work  by  Wickens  and  Yeh  (1997)  suggests  that  an  effectively  designed 
display  of  all  information  is  more  efficient  than  a  user-controlled  display  of  particular  information 
elements.  When  designing  displays,  an  effort  should  be  made  to  determine  if  there  are  “natural 
mappings”  (Helander,  1987)  between  a  type  of  meta- information  and  a  display  method  that  could  be 
exploited  to  make  the  presentation  of  meta-information  more  intuitive  and  therefore  minimize  potential 
overload  problems. 

6.3  Future  Research  Needs 

Our  analysis,  and  the  resulting  broad  conceptualization  of  meta-information  needs  in  C2  decision¬ 
making,  has  illuminated  a  number  of  areas  for  future  research.  In  particular,  there  remain  unanswered 
questions  about  the  impacts  of  different  forms  of  meta- information  on  specific  decision-making  tasks. 
Reasoning  should  be  examined  with  respect  to  the  particular  goal  or  task  and  the  types  of  meta¬ 
information  required,  and  additional  effort  is  needed  to  differentiate  reasoning  under  uncertainty  from 
reasoning  with  meta-information.  Computational  methods  are  needed  that  go  beyond  reasoning  about 
types  of  uncertainty  to  embrace  the  more  general  concept  of  meta-information  and  are  developed  with 
consideration  to  specific  information  and  meta-information  needs.  Finally,  significant  work  is  needed  in 
establishing  efficient  and  effective  ways  to  communicate  meta- information  to  the  commander;  research  is 
needed  to  understand  how  particular  visual  representations  and  interface  methods  are  more  or  less 
effective  for  different  types  of  meta- information.  In  all  of  these  cases,  we  believe  that  the  explicit 
discussion  of  meta-information  requirements  will  drive  the  design  of  more  effective,  safe,  and  efficient 
decision-support  systems. 
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General  Motivation 


o  Interest  in  uncertainty  w.r.t.  system  design  &  development 
o  In  Artificial  Intelligence  community 
o  Probabilistic  reasoning  techniques 
o  Representational  formalisms 
o  In  Cognitive  Engineering  community 

o  In  decision-making  (e.g.,  trust  and  uncertainty) 
o  For  visualization  and  interface  design 

o  In  Military  environments 
o  Asymmetric  warfare 
o  Increase  in  HUMINT 
o  Increase  in  information  in  NCW 

o  Anecdotes  across  many  domains... 
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General  Motivation 


o  "Uncertainty"  is  not  enough 

o  Information  may  be  qualified  in  other  ways 
Importance,  Quality,  Impact,  Pertinence 
o  Recency,  Staleness,  Timeliness 
o  Ambiguity,  Accuracy,  Precision 
o  Pedigree,  Confidence,  Reliability 


o 


o  "Meta-Information" 

...  is  a  concept/term  that  captures  information  qualifers  more 
generally 
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Goals 


o  Establish  that  meta-information  must  be  considered  in  C2 
system  design  by  summarizing  analyses  across  C2  domains 

o  Provide  guidelines  to  support  the  design  of  C2  decision-support 
systems  w.r.t.  meta-information 
o  Displays  and  user  interfaces 
o  Computational  methods 

o  Encourage  design  processes  that  aid  in  understanding  meta¬ 
information  requirements 

o  Because  of  task  and  context  dependence  of  meta-information 
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Analysis  Method 


o  Performed  analysis  as  part  of  Cognitive  Systems  Engineering 
methodology 


1 

'  Scenario  ' 

f 

Case  Studies 

Analysis 


Iterative  Development 


Deployment 
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o  Constructed  hypothetical  scenarios  to  explore  context 
o  Conducted  structured  interviews  with  domain  experts 
o  Performed  analysis  to: 

Identify  key  sources  of  complexity  and  types  of  decisions 

o  Uncover  sources  and  types  of: 
o  Data 
o  Meta-data 
o  Information 
o  Meta-information 

o  Identify  required  information  and  meta-information 
o  Discover  situational  influences  on  requirements 
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C2  and  C2-Related  Domains  Analyzed 


o  Intelligence  operations 
o  Small-unit  tactical  maneuvers 
o  Sensor  management 
o  Weather  impact  assessment 
o  Natural  disaster  management 


(Our  thanks  to  the  many  domain  experts  we  interviewed  and  observed!) 
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Results  of  Analysis 


o  Identified  information  needs 
o  Identified  meta-information  needs 
o  Defined  meta-information  concepts 


potentially  a  many-to-many  mapping 
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Working  Definitions 


o  Data :  output  (processed  or  unprocessed)  from  a  human  or  machine 
system  -  e.g., 

o  Acoustic  sensor  X  reports  34  Db 
o  Joe  says  it  is  raining 

o  Information :  an  input  to  a  directed  decision-making  process  -  e.g., 

A  storm  is  coming,  thus  I  will  not  launch  the  weather  balloon  until  tomorrow 

o  Meta -Data :  characteristics  or  qualifiers  of  data  -  e.g., 
o  Temperature  sensor  Y  has  an  error  of  +/-  0.1  deg  F 

o  Meta-Information :  characteristics  or  qualifiers  of  information, 
affecting  a  human's: 

Information  processing 
o  Situational  awareness 
o  Decision-making 

o  E.g.,  There  is  a  60%  chance  the  fire  is  located  at  {x,y>  therefore  I  will 
confirm  its  location  before  sending  fire  trucks 
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o  Is  "meta-information"  just  "information"?  No. 
o  It  qualifies  information 

o  It  may  be  reasoned  about  differently 

o  E.g.,  qualifiers  may  be  ignored  under  high  time  demands 
o  It  tends  not  to  be  regularly  captured  or  represented  in  many 
human-machine  systems  where  it  is  needed 

o  How  we  might  define  data,  meta-data,  information,  and  meta¬ 
information  depends  on 

o  The  decision-making  task 

o  The  context  or  situation 
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Meta-Information  Definitions 

across  Tasks 


Sensor  Management 

Tactical  Decision-Making 

Data 

Sensor  X  reports  42.2  dB 

Sensor  Y  reports  32.1°  F 

Sensor  X  reports  42.2  dB 

Sensor  Y  reports  32.1°  F 

Meta-Data 

Sensor  X  error  is  ±  .4  dB 

Sensor  Y  reports  at  5  Hz 

Sensor  X  error  is  ±  .4  dB 

Sensor  Y  reports  at  5  Hz 

Information 

Location  of  sensors 

Sensor  types 
“Health”  of  sensors 

Location  of  targets 

Type  of  targets 

Number  of  targets 

Meta-Information 

Accuracy  of  sensor  status 

Recency  of  sensor  status 

“Health”  of  sensors 

Coverage  of  sensors 

Accuracy  of  target  information 
Recency  of  target  information 
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Meta-Information  Definitions 

across  Tasks 


Sensor  Management 
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Data 

Sensor  X  reports  42.2  dB 
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Sensor  X  error  is  ±  .4  dB 
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Sensor  Y  reports  at  5  Hz 

Information 

Location  of  sensors 

Sensor  types 

“Health”  of  sensors 

Location  of  targets 

Type  of  targets 

Number  of  targets 
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More  Results 


o  Analyses  showed  wide  range  of  types  of  meta-information 

o  The  following  provide  examples  of  meta-information  types  we 
encountered... 
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Types  of  Meta-Information  Discovered 


o  Characteristics  of  the  source  of  the  information 
o  Type  of  data  the  source  can  produce 
o  Type  of  processing  used 
o  Range  of  data  generated 
o  Baseline  error  rates 
o  Frequency  of  reporting 

o  Ability  to  report  on  its  status  and  characteristics  of  that  report 
o  Inherent  biases 
o  Past  performance,  history 
o  Directly  observing  or  deriving  information 
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Types  of  Meta-Information  Discovered 


o  Characteristics  of  the  source  varying  with  other  information 
o  Time 

o  Location  in  environment  (e.g.,  terrain,  weather) 
o  Types  of  intermediate  processing 
o  Content  of  report 
o  Information  context 


o  Uncertainty 

o  Spatial  uncertainties 
o  Temporal  uncertainties 
o  Uncertainties  about  uncertainty  reporting 
o  Likelihood 
o  Probability 
o  Confidence 
o  Accuracy 
o  Precision 
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Types  of  Meta-Information  Discovered 


o  Ambiguity 

o  Specificity  or  resolution  of  information 
o  Level  of  abstraction  of  information 

o  Information  context  (i.e.,  relationship  to  other  information) 
o  Degree  of  confirming  or  disconfirming  information 

o  Paucity  of  information 

o  Frequency  of  reporting  of  information 
o  Missing  or  degraded  information  qualifiers 
o  Information-to-noise  ratio 
o  History 

o  Reliability  of  source 

o  W.r.t.  source  characteristics 
o  W.r.t.  information  context 
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Types  of  Meta-Information  Discovered 


o  Credibility  of  content  from  source 
o  W.r.t.  reliability 
o  W.r.t.  type  of  content 
o  W.r.t.  type  of  source 
o  W.r.t.  information  context 

o  Relevance  or  pertinence 

o  W.r.t.  specific  mission  goals 
o  W.r.t.  actual/perceived  information  needs 
o  W.r.t.  broader  operational  context 
o  W.r.t.  current  hypotheses  about  the  situation 

o  Temporal  qualifiers 
o  Staleness 
o  Recency 

o  Certainty  about  time  of  reporting 
o  Latency 
o  Lag 

o  Absence  of  expected  information 
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Meta-Information  and 
C2  Decision-Making 


o  Uncovered  three  complexities  related  to  decision-making  and 
meta-information 

o  Failure  to  recognize  relevant  meta-information 
o  Failure  to  process  meta-information  appropriately 
o  Failure  to  properly  utilize  meta-information 

o  These  complexities  apply  to  both 
o  Fluman  decision-making 
o  Machine  reasoning 
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Implications  &  Future  Work: 
Computational  System  Design 


o  Need  to  represent  meta-information  needs  in  data  structures, 
computational  processes 

o  Need  to  calculate  meta-information  from  data  and  meta-data 
o  Need  to  aggregate  meta-information 
o  Need  to  process  types  of  meta-information  simultaneously 
o  Need  to  minimize  impact  of  additional  computation 

o  Future  work: 

What  representational  formalisms  are  amenable  to  handling 
multiple  types  of  qualifiers? 

What  computational  processes  support  reasoning  over  qualified 
information? 

To  what  extent  can  existing  methods  be  adapted  to  support  meta¬ 
information  needs? 
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Implications  &  Future  Work: 
Display  and  Interface  Design 


o  Need  to  communicate  meta-information  in  a  situation-  and 
task-  relevant  manner 

What  visualization  methods  work  for  what  types  of  meta¬ 
information? 

How  does  the  information  type  and  its  display  method  interact  with 
the  meta-information  visualization? 

o  Need  to  avoid  overloading  the  user  with  the  presentation  of 
meta-information 

How  and  when  does  the  presentation  of  meta-information  cause 
overload? 

o  What  user  interface  mechanisms  could  aid  in  avoiding  overload? 

o  Need  to  aid  reasoning  about  and  with  meta-information 

What  displays/UIs  facilitate  a  user's  ability  to  understand  and 
exploit  meta-information? 

o  How  can  users  be  trained  to  recognize  and  use  meta-information? 
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Conclusions 


o  We  must  go  beyond  thinking  only  about  uncertainty 
o  Information  may  be  qualified  in  many  ways 

o  Meta-information  needs  should  be  reflected  in  C2  decision 
support  systems: 

o  As  part  of  underlying  computational  methods 
o  As  part  of  displays  and  interfaces 
o  With  awareness  of  task  and  situation  dependencies 

o  Additional  work  remains  to  be  done... 
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